# 


UNITED STATES PATENT APPLICATION FOR 


MODULAR SYSTEM 
WITH SYNCHRONIZED TIMING 


(3 
\D 

m 
in 

w 

h- 

ru 

ill 

m 

3 1 

O 


Inventors : 
Stanley P. Woods 
Hans J. Sitte 
Bruce Hamilton 


CERTIFICATE OF MAILING BY "EXPRESS MAIL" 
UNDER 37 C.F.R. § 1.10 

"Express Mail" mailing label number: EI5609874 09US 
Date of Mailing: 5-9-2001 

I hereby certify that this correspondence is 
being deposited with the United States Postal 
Service, utilizing the "Express Mail Post Office to 
Addressee" service addressed to Assistant 
Commissioner for Patents, Washington, D.C. 20231 and 
mailed on the above Date of Mailing with the above 
"Express Mail" mailing label number. 


Paul H. Horstmann, Reg. No. 36,167 
Signature Date: 5 "9" ZOO \ 


BACKGROUND OF THE INVENTION 


Field of Invention 

The present invention pertains to the fields of 
measurement/control systems. More particularly, this 
invention relates to a modular system with 
synchronized timing . 

Art Background 

A wide variety of systems including 
measurement/control systems commonly include a 
variety of instruments that provide measurements and 
apply stimulus. Examples of instruments are numerous 
and include instruments for measuring physical 
characteristics such as temperature, pressure, 
voltage, etc., as well as the instruments for 
generating stimuli such as heat, mechanical stimuli, 
chemical stimuli, electrical stimuli, etc. 

It is often desirable in a system to provide 
coordination of the measurements obtained by its 
instruments and/or the stimuli applied by its 
instruments. For example, it may be desirable to 
precisely control the timing of stimuli applied by an 
instrument and/or accurately record the time at which 
measurements are obtained by the instrument. In 
addition, it is often desirable that a system have a 
modular design in which different types of 
measurement modules and stimulus modules may be 
employed . 

Prior systems commonly employ modular 
instruments which usually have an instrument housing 
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that holds a set of modules each of which performs a 
measurement and/or stimulus function. Typically, the 
modules in such an instrument housing communicate via 
an internal bus of the instrument housing. Examples 
5 of internal buses include VME, VXI, PXI, PCI, and 

CPCI. One example of a such a bus implemented with a 
set of cables is the IEEE 488 standard. Typically, 
the application of stimuli and/or the measurements 
obtained by such an instrument are coordinated by 
10 transferring commands via its control bus or by using 
dedicated external trigger wires. 

Typically, such an instrument housing accepts up 
to a maximum number of modules. If a system requires 

15 a number of modules that exceeds the maximum for an 
instrument housing then additional instrument 
housings must usually be employed. The use of 
multiple instrument housings usually requires an 
engineering effort to provide coordination among the 

20 modules contained in different instrument housings. 

Unfortunately, such an engineering effort typically 
increases the costs associated with using prior 
systems . 

In addition, the modules of prior systems are 
25 typically confined to an instrument housing or a 

relatively short distance from an instrument housing 
and have transducer connections of limited length. 
As a consequence, such prior systems are usually not 
suitable for obtaining coordinated measurements and 
30 applying coordinated stimuli at points of a system 
which are widely dispersed. 
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SUMMARY OF THE INVENTION 


A modular system is disclosed with a set of 
modules having synchronized timing. The synchronized 
timing of the modules enables precise coordination of 
measurements and stimuli for an arbitrary number of 
modules. The modules communicate and maintain time 
synchronization using a communication mechanism that 
may be adapted to localized positioning of modules 
and/or widely dispersed positioning of the modules. 
Coordination of measurements and/or stimuli may thus 
be accomplished for systems which span small or large 
distances, and which have few modules or more than 
can fit in an enclosure. This coordination is 
accomplished with no change to the underlying 
functionality in the modules or the method of using 
the modules. This avoids substantial modification to 
the software in the modules and avoids substantial 
modifications to higher level application software 
that uses the modules. 

Other features and advantages and applications 
of the present invention will be apparent from the 
detailed description that follows. 
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BRIEF DESCRIPTION OF THE DRAWINGS 


The present invention is described with respect 
to particular exemplary embodiments thereof and 
reference is accordingly made to the drawings in 
which : 

Figure 1 shows a system according to the present 
teachings with a relatively localized positioning of 
modules ; 

Figure 2 shows a system according to the present 
teachings which is suitable for a more dispersed 
positioning of modules; 

Figure 3 shows a system according to the present 
teachings which is suitable for greatly dispersed 
modules ; 

Figure 4 shows an example embodiment of a module 
according to the present teachings. 
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DETAILED DESCRIPTION 


Figure 1 shows a system 10 according to the 
present teachings. The system 10 includes a set of 
modules 20-24 each of which is adapted to obtain a 
measurement or apply a stimulus in the system 10. 
Examples of a measurement are numerous and include 
voltmeter functions, oscilloscope functions, logic 
analyzer functions, obtaining digital inputs, 
obtaining digital inputs which are trigger outputs 
from other devices, reading a counter, a 
communication protocol analyzer, and network 
analyzer. Examples of a stimulus are also numerous 
and include digital outputs, digital outputs used to 
trigger other devices, signal generator functions, an 
arbitrary waveform generator, a digital data 
generator, and a pulse generator. In addition, the 
instrument function may be a switch matrix used to 
control the routing of measurement and stimulus 
signals . 

The system 10 includes a communication device 12 
that enables communication among the modules 20-24. 
The communication device 12 is selected to provide 
localized and/or widely dispersed communication among 
the modules 20-24. 

Each module 20-24 includes mechanisms for 
communication according to a communication protocol 
associated with the communication device 12. The 
communication protocol is preferably a packet-based 
protocol. In one embodiment, the communication 
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protocol is Ethernet and the communication device 12 
may be an Ethernet hub, repeater, or switch. 

Each module 20-24 includes a synchronized clock 
5 that synchronizes its time with the times in the 

synchronized clocks of the other modules 20-24. In 
one embodiment, the synchronized clocks in the 
modules 20-24 perform time synchronization by 
measuring the transmit and receive time of timing 
10 packets transferred via the communication device 12 

and communicating these measured times to each other. 

□ 

CO In one embodiment, synchronization is performed 

Wl . 

m 15 in the modules 20-24 using a synchronization protocol 

^ described in U.S. Patent no. 5,566,180. Other 

r|| possible mechanisms for time synchronization among 

jL the modules 20-24 include network time protocol 

y| (NTP) , global positioning system (GPS) receivers, 

f *\ 

**j 20 polling conducted by a master clock, and the periodic 
p broadcast of time to the modules 20-24. 

Each module 20-24 includes mechanisms for 
performing a measurement function or a stimulus 

25 function. For example, if the module 20 is a 
temperature sensor then it may include the 
appropriate hardware/software for sensing temperature 
and for generating digitized values that indicate the 
sensed temperature. In another example, if the 

30 module 20 is a signal generator then it may include 
the appropriate circuitry/software for generating a 
signal. In some embodiments, some of the measurement 
hardware or stimulus hardware, and/or digitizing 
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hardware may be implemented in a subsystem to which 

the module 20 may be connected. For example, one or 

more of the modules 20-24 may be connected to an 
appropriate sensor or actuator. 

The embodiment of the system 10 shown in Figure 
1 is one in which the modules 20-24 plug into an 
instrument bay 11 that holds the communication device 
12 and a power supply 14. The physical communication 
links between the modules 20-24, for example Ethernet 
lOBaseT lines, are contained in the instrument bay 
11. This embodiment is adapted to applications in 
which the points of a system which are to be measured 
and/or stimulated are in a relative close spatial 
proximity . 


One or more of the modules 20-24 may be removed 
and operated outside of the instrument bay 11 with no 
change to the underlying communication and time 
synchronization or methods of coordinating 
measurements and/or stimuli among the modules 20-24. 
For example, the module 23 is located outside of the 
instrument bay 11. The module 23 is connected to the 
instrument bay 11 via a set of wires 13 that provide 
power lines and communication lines. The power line 
portion of the wires 13 and the communication line 
portion of the wires 13 may be contained in the same 
cable or in separate cables. 


Figure 2 shows a system 40 according to the 
present teachings for applications in which the 
points of the system 40 which are to be measured 
and/or stimulated may be spaced farther apart. The 
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system 40 includes a communication device 42 and a 
set of modules 50-54 which include synchronized 
clocks and mechanisms for applying a stimulus and/or 
performing a measurement and for performing 
communication via the communication device 42. 
Though not shown, the system 40 includes an 
instrument bay housing with a power supply. 

The modules 50-54 communicate with the 
communication device 42 via a set of physical 
communication links 30-34. The physical 
communication links 30-34 have a length adapted to 
the particular spatial arrangement of the modules 50 
54 and a physical implementation which is adapted to 
the communication protocol of the communication 
device 42. 

Figure 3 shows a system 60 according to the 
present teachings for applications in which the 
points of the system 60 which are to be measured 
and/or stimulated may be spaced at great distances 
apart. The system 60 includes a set of modules 70-7 
that communicate via a communication device 62 and a 
set of modules 80-84 that communicate via a 
communication device 64 . The communication devices 
62-64 may each be contained in an instrument bay 
housing that includes a power supply. 

The communication devices 62 and 64 communicate 
via a communication network 100. The communication 
network 100 represents an infrastructure of physical 
communication links and hardware/software for widely 
dispersed communication. The communication network 
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100 enables transfer of communication packets among 
the modules 70-74 and 80-84 including the timing 
packets used for clock synchronization by the modules 
70-74 and 80-84 and event packets associated with 
measurement events and stimulus events. The 
communication network 100 may be a packetized network 
such as Ethernet or a network such as LonTalk which 
is adapted to control systems. 

Numerous other arrangements for a system with 
modules having synchronized clocks according to the 
present techniques are possible. For example, the 
communication network 100 may be a sub-net of a 
larger network which is connected to other modules 
having synchronized clocks. In one embodiment, the 
protocol for clock synchronization among the modules 
does not cross sub-nets. In embodiments that include 
multiple sub-nets each sub-net may be provided with a 
mechanism for clock synchronization such as GPS time. 
The GPS time may then be used in a master clock to 
drive clock synchronization in the modules on each 
sub-net. Alternatively, specialized routers that 
function as both master and slave of the time 
synchronization protocol used in the modules may be 
employed to propagate the synchronization protocol 
across sub-nets. 

Figure 4 shows one embodiment of the module 50. 
The modules 20-24, 52-54, 70-74, and 80-84 may be 
implemented in a substantially similar manner. The 
module 50 includes a communication interface 200, a 
synchronized clock 202 and a set of instrument 
hardware 204. 
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The communication interface 200 enables 
transmission and reception of messages in packets via 
the physical communication link 30. A timing data 
packet 118 and a follow up packet 116 carried on the 
physical communication link 30 are packets used for 
time synchronization. An event packet 120 carried on 
the physical communication link 30 is associated with 
the measurement or stimulus behavior of the 
instrument hardware 204. 


The communication interface 200 provides the 
received timing data packet 118 and the received 
follow up packet 116 to the synchronized clock 202. 
The timing data packet 118 and the follow up packet 
116 are generated by a master clock in one of the 
other modules with which the module 50 communicates. 
Alternatively, the master clock may be contained in a 
controller or computer system or communication device 
which is reachable via the physical communication 
link 30. The master clock may be a real-time clock. 


The timing data packet 118 includes a delimiter 
154 that identifies it as a timing data packet for 
the synchronization protocol. The follow up packet 
116 includes a time stamp 150. The time stamp 150 
indicates the local time in the master clock when the 
timing data packet 118 was generated. 

The synchronized clock 202 includes a time 
packet recognizer and a clock and a latch. The time 
packet recognizer detects a unique timing point in 
the recovered bit stream for the timing data packet 
118. Upon detection of the unique timing point, the 
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time packet recognizer causes the latch to latch a 
time value from the clock. The time value held in 
the latch indicates the local time at which the time 
packet recognizer received the timing data packet 
5 118. Thereafter, the time packet recognizer receives 
the follow up packet 116 and extracts the time stamp 
150. The difference between the time stamp 150 and 
the time value in the latch indicates the relative 
synchronization of the master clock and the 
10 synchronized clock 202. Once this difference is 

computed the time packet recognizer uses it to adjust 
the time value in the clock inside the synchronized 

a 

clock 202 to conform it to the master clock. 

«i 

in 

|t| 15 The adjustment of the time value in the clock in 

the synchronized clock 202 may be accomplished by 
jfU implementing the clock as a counter driven by an 

* H oscillator with sufficient stability and resolution 

If] given the timing precision desired. The least 

n 

20 significant few bits of the counter may be 
O implemented as an adder so that an increment of the 

counter on oscillator periods may be occasionally 
increased or decreased to effectively speed up or 
slow down the clock in accordance with the results of 

25 the computation of the difference between the time 
stamp 150 and the time held in the latch. 

The communication interface 200 provides the 
received event, packet 120 to the instrument hardware 
30 204. The instrument hardware 204 decodes a set of 
event data 160 in the event packet 120 and carries 
out a measurement or stimulus function accordingly. 
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If the module 50 performs a measurement function 
then the event data 160 may include a command to 
start obtaining measurements or to stop obtaining 
measurements or to obtain measurements at specified 
times or at specified time intervals. In response, 
the instrument hardware 204 obtains measurements via 
a probe 212. The instrument hardware 204 obtains a 
synchronized time 210 from the synchronized clock 202 
and may use it to trigger a measurement or to time- 
stamp a measurement once obtained. The instrument 
hardware 204 may generate messages that contain 
measurements obtained and associated time-stamps and 
send the messages via the physical communication link 
30. A computer system or controller or other 
instrument may gather these messages and extract the 
obtained measurements. 


If the module 50 performs a stimulus function 
then the event data 160 may include a command to 
apply the stimulus at a specified trigger time. In 
response, the instrument hardware 204 triggers the 
application of the stimulus via the probe 212 by 
comparing the synchronized time 210 from the 
synchronized clock 202 to the trigger time from the 
event data 160. 


Coordination in a system may be provided by 
event packets transferred to the modules to set up 
the measurement or stimulus functions. The event 
packets may specify times at which event functions 
are to be triggered. The modules that time-stamp 
measurements may provide return packets that contain 
the measured data and time-stamps. The fact that the 


Attorney Docket No. 10003186 


synchronized clocks in the modules maintain time 
synchronization with respect to one another yields 
precise coordination of event triggering and precise 
time-stamps of measurement events. 

The instrument hardware 204 may include a 
processor and associated software for performing an 
application-specific measurement or stimulus 
functions. The processor may also execute software 
for the communication protocol associated with the 
communication interface 200. For example, the 
processor may execute communication protocol layers 
associated with Ethernet communication. 


The view of the software coordinating 
measurements, stimuli, or measurement results using 
module 50 is the same whether the module is installed 
in an instrument bay as in the embodiment shown in 
Figure 1 or is dispersed as in one of the embodiments 
shown in Figures 2-3. For example, the same 
communication protocol is used for all these 
embodiments with the same packet format. This 
consistent software view enables flexible placement 
of the modules with minimal engineering effort for 
adapting the modules to different physical 
configurations . 


The modules described above provide a stand- 
alone measurement or stimulus function like miniature 
instruments while providing the capability of being 
grouped together into multiple channels in one 
location or across wide distances. As a consequence, 
the range of applications for the modules is large. 
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The modules provide for relatively easy coordination 
using their synchronized clocks. 

Each module according to the present teachings 
5 contains an entire instrument from input and/or 

output with communications and time synchronization 
capability. The modules communicate and/or interact 
via an external network which may be user accessible 
and with or without a controller or computer. 
10 Whether the modules are contained in the same 

physical case, in near proximity, or whether they are 
widely dispersed over a large geographic area 
■J) (including but not limited to the Earth) is no longer 

a concern of the instrument user. 


ul 

Ul 15 


see 


A system with modules according to the present 
fU teachings may be employed in data acquisition 

*» systems, distributed control systems, supervisory 

Ul control and data acquisition systems, and programable 

b p 20 logic controllers, to name a few examples. 
O 

According to the present teachings, coordinated 
application of stimuli and/or coordinated 
measurements may be achieved by sending a message to 

25 each module which includes an identification of which 
stimulus to apply or measurement to obtain and a time 
at which the stimulus is to be applied or the 
measurement obtained. This method of coordinated 
stimuli/measurement is substantially unchanged in all 

30 of the module configurations disclosed above. This 
avoids substantial re-engineering when instruments 
are added, removed, or changed in the system as is 
common. This is in stark contrast to prior systems 
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in which coordinated measurements and/or stimuli is 
achieved by requesting a predefined measurement 
and/or stimulus from a controller or by writing a 
special-purpose program and running on the 
controller. If prior instruments are connected to 
separate controllers then the system must be re- 
engineered so that its operation does not require 
coordinated application of measurements and/or 
stimuli . 

According to the present teachings, measurements 
which are made by different modules (possibly in an 
uncoordinated manner) may be correlated by having 
each module report its result along with the time at 
which it took the measurement (the time-stamp) . It 
may then be determined that if two measurements have . 
equal time-stamps they were taken at the same time. 
If their time-stamps differ it can be determined 
which was taken first and how much time elapsed 
between measurements. This benefit is substantially 
unchanged in all of the module configurations 
disclosed above and helps avoid re-engineering in 
response to changes in the system. This is in stark 
contrast to prior systems in which measurements are 
correlated according to their sequence of arrivals. 
In such systems, it cannot be determined precisely 
when the measurements were made. As such a prior 
system gets physically larger, the delay between a 
measurement and the arrival of its result gets larger 
and increases uncertainty about the precise time of 
the measurement. 
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The foregoing detailed description of the 
present invention is provided for the purposes of 
illustration and is not intended to be exhaustive or 
to limit the invention to the precise embodiment 
5 disclosed. Accordingly, the scope of the present 
invention is defined by the appended claims. 
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